Multi-Display Alignment Through Observed Interactions

ABSTRACT

In some implementations, a computing device can perform multi-display alignment through observed user interactions. The computing device can receive user input aligning a first alignment object on a first display device with a second alignment object on a second display device. The computing device can align the display buffers for each display device based on the positions of the alignment objects in each display buffer corresponding to each display device. The computing device can align display buffers based on observed movements of graphical objects between multiple display devices. When display buffers corresponding to the display devices are misaligned, the user may correct the path of a graphical object when moving the graphical object between display devices. The computing device can detect the correction and align the display buffers of the display devices so that graphical objects are presented at the appropriate locations when moved between the display devices.

RELATED APPLICATIONS

This application claims the benefit of the filing date of U.S.Provisional Patent Application No. 62/907,197 filed on Sep. 27, 2019,which is hereby incorporated by reference.

TECHNICAL FIELD

The disclosure generally relates to aligning multiple displays inmultiple display systems.

BACKGROUND

Often users of computing devices will have multiple display devicesconnected to their computing devices. For example, a user who works on amobile computer (e.g., laptop computer, tablet computer, etc.) having asmall display will often connect to a larger external display,additionally it is increasingly common for a portable device such as amobile tablet computer to be used as an additional display for anothersystem (such as Apple's Sidecar feature for iPads to be used with Macdesktop or Laptops). A user who works from a desktop computer mayconnect the desktop computer to multiple external displays. Theseadditional displays may be connected via a cable or wirelessly, and,being hand held, their physical position, and thus expected logicalarrangement with the other displays might change dynamically. Thisadditional display area makes using the computing device much easier andmore efficient when simultaneously working with multiple documentsand/or multiple software applications on the computing device.

When connecting a second or subsequent display device to a computingdevice that is already connected to a first display device, the userwill often be required to explicitly configure the physical arrangementof the multiple display devices relative to each other so that when theuser moves a cursor, window, or other graphical object, from one displaydevice to another, or leaves an object spanning across multipledisplays, the object moves smoothly and continuously from display todisplay as expected (e.g., the object moves as if it were a physicalobject leaving one edge of one display and appearing on the physicallyadjacent display along the path of the cursor movement at the expectedlocation on the destination display device) and/or the spanning objectis aligned as expected across the multiple displays. In one of thetypical ways of configuring the relative positions and/or alignment ofmultiple displays in a multiple display system, the computing device canpresent on one of the display devices graphical representations of themultiple display devices connected to the computing device. The user canthen provide input to arrange the graphical representations of thedisplay devices such that the representations mimic the physicalalignment and/or relative positions of the actual, physical displaydevices. The computing device can then use the arrangement of thegraphical representations of the display devices to determine how toalign the display buffers for the display devices and/or how move acursor from one display to another.

SUMMARY

In some implementations, a computing device can perform multi-displayalignment through observed user interactions. The computing device canreceive user input aligning a first alignment object on a first displaydevice with a second alignment object on a second display device. Withknowledge of the displays' characteristics such as resolution, pixelsdensity, physical size, height of display off table surface, thecomputing device can use the observed interaction data to solve for andalign the display buffers for each display device based on the positionsof the alignment objects in each display buffer corresponding to eachdisplay device. The alignment itself may consist of scaling, rotation,and translation operations providing a transformation from a large,continuous, virtual rendering space to the pixels in the individualdisplays. Alternately the alignment might be used to translate thecursor and other graphical objects from display to display in aphysically analogous manner without requiring a continuous extendeddesktop to exist.

In some implementations, the computing device can align display buffersbased on observed movements of the use attempting to move graphicalobjects between multiple display devices. When the user intends to movean object from one display to another, they will naturally drag theobject in the direction of where they want to place it on the seconddisplay thus providing valuable information regarding the physicalrelationship between the displays otherwise unknown to the system. Whendisplay buffers corresponding to the display devices are misaligned, theobject may leave the first display where the user dragged it, but thesystem may cause the object to emerge on the second display in alocation different from where the user anticipated potentially causingthe user to correct the path of the graphical so as to reach theintended target on the new display. The computing device can detect thecorrection, as a deviation in trajectory after the object transferred tothe second display from the user's initial arc on the first display, andalign the display buffers of the display devices so that graphicalobjects are presented at the appropriate locations when moved betweenthe display devices.

Particular implementations provide at least the following advantages.Multiple display devices (e.g., the display buffers thereof) can be moreaccurately or precisely aligned thereby improving the user experiencewhen interacting across multiple displays. Multiple display devices canbe dynamically aligned without requiring the user to perform a specificor explicit alignment process or interaction. Display alignments can berefined over time by analyzing multiple user interactions acrossmultiple display devices. Changes in display alignment due to displayangle, laptop positioning, the user's position, or other changes, can bedynamically detected and alignment adjustments made without the userhaving to perform a prescribed alignment interaction or input. Bydynamically monitoring interaction, re-solving for position, aligningthe display buffers, and cursor/object transformations in a multipledisplay system, the system can improve the user experience by presentinggraphical objects at user expected locations when moving graphicalobjects across multiple display devices.

Details of one or more implementations are set forth in the accompanyingdrawings and the description below. Other features, aspects, andpotential advantages will be apparent from the description and drawings,and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an example system for multi-displayalignment through observed user interactions.

FIG. 2 is a diagram illustrating aligning display buffers for multipledisplay devices.

FIG. 3 is a diagram illustrating a display buffer alignment techniqueusing a display alignment model.

FIG. 4 is a diagram illustrating aligning display buffers based on aforced user interaction.

FIG. 5 is a diagram illustrating various relative display alignments.

FIG. 6 illustrates an example graphical user interface for provokingmultiple display interaction to determine display alignment.

FIG. 7 illustrates an example graphical user interface for observingmultiple display interactions to determine display alignment.

FIG. 8 is a diagram illustrating a cross-display movement pathindicative of misaligned display buffers.

FIG. 9 is a diagram illustrating alignment of display buffers based onobserved cross-display interactions.

FIG. 10 is a diagram illustrating alignment of multiple external displaybuffers based on observed cross-display interactions.

FIG. 11 is a diagram illustrating alignment of multiple external displaybuffers based on observed cross-display interactions.

FIG. 12 is flow diagram of an example process for aligning displaybuffers based on a forced user interaction.

FIG. 13 is a flow diagram of an example process for aligning displaybuffers based on observed interactions.

FIG. 14 is a block diagram of an example computing device that canimplement the features and processes of FIGS. 1-13.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

The disclosure generally relates to aligning multiple displays inmultiple display systems into a common extended desktop space, orindependent spaces, both potentially transferring the cursor fromdisplay to display based on the display's physical size and location sothat the cursor logically and visually continuously tracks movementjumping from display to display according to the physical location(similar to a user pointing a laser pointer directly at the display).However, to achieve this, many pieces of precision information arerequired including: the number of pixels on the display, the specialpixel density (might be asymmetric too), the height of the display offthe desktop, the physical location and orientation of the displays.While there exist primitive mechanisms for users to describe the displayarrangement relationships to the system, they are cumbersome andimprecise leading to less than optimal results which then deteriorate asthe physical positioning changes over time without the user fine-tuningthe arrangement process. This disclosure describes a mechanism toautomatically glean and fine tune these relationships by observing howthe user interacts with the system.

FIG. 1 is a block diagram of an example system 100 for multi-displayalignment through observed user interactions. In some implementations,system 100 can include user device 102. For example, user device 102 canbe a computing device, such as a desktop computer, laptop computer,tablet computer, laptop computer, smartphone, smartwatch, wearabledevice, or any other type of computing device.

In some implementations, system 100 can include display device 110. Forexample, display device 110 can be a physical display device connectedto user device 102. Display device 110 can be integral to user device102, such as a built-in display of a laptop computer, smartphone, tabletcomputer, etc. Display device 110 can be an external display connectedto user device 102 by a physical connection, such as a video cable thatconnects a monitor to a desktop computer. Display device 110 can be theprimary display device for user device 102, for example.

In some implementations, system 100 can include display device 130. Forexample, display device 130 can be an external display device, such asan external monitor, television, projector, augmented reality display,virtual reality display, headset display, smart glasses, smart watch, orother display device. Display device 130 can be connected to user device102 through network 120. For example, network 120 can be a wired (e.g.,ethernet) or wireless (e.g., Wi-Fi, Bluetooth, peer-to-peer, etc.)network. Network 120 can represent a direct cable connection betweenuser device 102 and display device 130. Thus, display device 130 can beconnected to user device 102 through a wired or wireless connection.

In some implementations, user device 102 can include display controller104. For example, display controller 104 can manage the presentation ofdata, graphics, etc., on one or more connected display devices (e.g.,display device 110, display device 130, etc.) connected to user device102. Display controller 104 can manage a display buffer (e.g., framebuffer, screen buffer, video buffer, etc.) for each display deviceconnected to user device 102. When presenting video (e.g., includinggraphical objects, graphical user interfaces, etc.), user device 102and/or display controller 104 can store data (e.g., a bit map, a pixelmap, etc.) defining a video frame to be presented on a display device inthe display buffer corresponding to the display device. Frames of videocan be generated based on the data in the display buffer and sent to thecorresponding display device for presentation, according to well-knownmechanisms. For example, display controller 104 can manage displaybuffer 112 for display device 110. Display controller 104 can managedisplay buffer 132 for display device 130. Each display buffer can beconfigured according to the specifications of the corresponding displaydevice. For example, display buffer 112 can be configured based on thedisplay capabilities, resolution, screen size, pixel dimensions, etc.,of display device 110. Display buffer 132 can be configured based on thedisplay capabilities, resolution, screen size, pixel dimensions, etc.,of display device 130.

When user device 102 (e.g., the operating system, an application, etc.)presents a graphical element (e.g., graphical user interface, text,images, icons, cursor, etc.) on a display, user device 102 can write thegraphical element to the appropriate location (e.g., horizontal,vertical pixel location) in the appropriate display buffer. The data inthe display buffer can then be sent to the corresponding display devicefor presentation to the user. By changing the location (x,y pixellocation) of the graphical element over time, user device 102 can causethe graphical element to be animated on the corresponding displaydevice. Thus, when a user provides input to move a graphical object(e.g., cursor, alignment object, window, etc.), user device 102 and/ordisplay controller 104 can, in response to the input, change thelocation of the graphical object in the display buffer according to theuser input.

When multiple displays are connected to user device 102, the user maywish to move a cursor and/or other graphical object (e.g., applicationwindow, folder, etc.) between the displays. To enable a smoothtransition between display devices 110 and display device 130, displaybuffer 112 and display buffer 132 can be associated with displayalignment data that describes how display buffer 112 and display buffer132 should be positioned (e.g., left/right, above/below) and/or aligned(e.g., horizontal offset, vertical offset, spacing between displays,etc.) with respect to each other. When configured correctly, thisalignment data should correspond to, or represent, how the physicaldisplay devices 110 and display device 130 are positioned with respectto each other in the real-world environment.

In some implementations, user device 102 can be configured with defaultalignment data (e.g., including display specifications, typicalpositioning, typical alignment, etc.) for various models of displaydevices that may be connected to user device 102. When user device 102connects to a display device (e.g., display device 120, display device130, etc.), user device 102 can receive model identification informationfrom the display device and use the model identification information toobtain default alignment data for the connected display device.

The default alignment data for each display device (e.g., display device110, display device 130, etc.) can indicate the display dimensions(e.g., 8 inches wide×8.94 inches high), the native resolution (e.g.,2560-by-1600 pixels native resolution), the pixel density (e.g., 227pixels per inch), the size of the bezels (ie the distance between theaddressable display and the physical edge of the enclosure, the defaultspacing relative to other displays (e.g., 2 inches with the displaysdisplays almost physically touching), the default horizontal or verticalalignment offset relative to other displays or relative to a surfaceupon which the display is sitting (e.g., 3 inches off the restingsurface), and/or a default directional position relative to otherdisplays (e.g., above, below, right, left). For example, the defaultalignment data for display device 110 may indicate that display device110 is an integral display of a laptop computer that is 9×8 inches, 227pixels per inch, 3 inches off the resting surface, and should bepositioned to the right of other displays. The default alignment datafor display device 130 may indicate that display device 130 is astandalone display that is 18×10 inches, 180 pixels per inch, and 5inches off the resting surface. Based on this default alignment data,display controller 104 can determine the relative positions of displaydevice 110 and display device 130 and their relative alignment (e.g.,display device 110 is to the right of display device 130 and has a −50pixel vertical offset).

FIG. 2 is a diagram 200 illustrating aligning display buffers formultiple display devices. As described above, display controller 104 candetermine the relative positions of display buffer 112 and displaybuffer 132 based on default alignment data. Continuing the example aboveand represented by configuration 202 in diagram 200, display controller104 can determine that the left edge of display buffer 112 is proximateto (e.g., but spaced apart from) the right edge of display buffer 132because the alignment data for display device 110 indicates that displaydevice 110 should be positioned to the right of display device 130.Display controller 104 can determine the relative vertical alignment ofdisplay buffer 112 and display buffer 132 based on default alignmentdata. For example, based on the display dimensions, resolution, and/orpixel density data in the default alignment data for display device 110and display device 130, display controller 104 can determine therelative vertical alignment display buffer 112 and display buffer 132.For example, display controller can determine that a specific row ofpixels (Y2) in display buffer 112 aligns with (e.g., is at the sameheight as) a specific row of pixels (Y1) in display buffer 132. Thus, ifY1 is 50 and Y2 is 100, the vertical offset value between display buffer132 and display buffer 112 is 50. For example, when moving graphicalobjects from display buffer 112 to display buffer 132, displaycontroller 104 can subtract 50 from the pixel row index of the locationof the graphical object when it leaves display buffer 112 to determinethe pixel row placement of the graphical object when it enters displaybuffer 1321. When moving graphical objects from display buffer 132 todisplay buffer 112, display controller 104 can add 50 to the pixel rowindex of the location of the graphical object when it leaves displaybuffer 132 to determine the pixel row placement of the graphical objectwhen it enters display buffer 112.

In other configurations (not shown), display controller 104 maydetermine that display buffer 112 is to the left of display buffer 132and determine corresponding rows of pixels (e.g., Y1, Y2) in displaybuffer 112 and display buffer 132 that align display buffer 112 anddisplay buffer 132 vertically. Display controller 104 can then determinethe appropriate vertical offset for moving graphical objects betweendisplay buffers (e.g., display devices). For example, display controller104 can apply the appropriate vertical offset to determine where thegraphical object should appear when moved from one display to another.

In yet other configurations (e.g., configuration 204), displaycontroller 104 may determine that display buffer 112 should be placedbelow (or above) display buffer 132 and determine corresponding columnsof pixels (e.g., X1, X2) in display buffer 112 and display buffer 132that align display buffer 112 and display buffer 132 horizontally.Display controller 104 can then determine the appropriate horizontaloffset for moving graphical objects between display buffers (e.g.,display devices). For example, display controller 104 can apply theappropriate horizontal offset to determine where the graphical objectshould appear when moved from one display to another.

As described above, after determining the relative positions of each ofthe display devices and/or display buffers in a multiple display systembased on the default alignment data, display controller 104 candetermine a horizontal and/or vertical alignment offset values for thedisplay buffers that can be used to determine where to present graphicalobjects (e.g., cursor, windows, etc.) that are moved between displaydevices. For example, with reference to configuration 202 in diagram200, Y1 may correspond to pixel row 1000 of display buffer 132 while Y2may correspond to pixel row 800 of display buffer 112. Thus, displaycontroller 104 may determine a vertical alignment offset of −200 fortransitioning, or moving graphical objects, from display buffer 132(e.g., display device 132) to display buffer 112 (e.g., display device110). For example, when moving a cursor from pixel row 900 of displaybuffer 132 to display buffer 112, display controller 104 can determinewhere to place the cursor in display buffer 112 by subtracting 200 from900 and placing the cursor at pixel row 700 at the left edge of displaybuffer 112. When moving the cursor from display buffer 112 to displaybuffer 132 the negative of the vertical alignment offset can be used todetermine where to place the cursor in display buffer 132.

As another example, with reference to configuration 204 in diagram 200,X1 may correspond to pixel column 1200 of display buffer 132 while X2may correspond to pixel column 200 of display buffer 112. Thus, displaycontroller 104 may determine a horizontal alignment offset of −1000 fortransitioning, or moving graphical objects, from display buffer 132 todisplay buffer 112. For example, when moving a cursor from pixel column1100 of display buffer 132 to display buffer 112, display controller 104can determine where to place the cursor in display buffer 112 bysubtracting 1000 from 1100 and placing the cursor at pixel column 100 atthe top edge of display buffer 112. When moving the cursor from displaybuffer 112 to display buffer 132 the negative of the horizontalalignment offset can be used to determine where to place the cursor indisplay buffer 132.

After determining the relative horizontal (e.g., right/left) or vertical(top/bottom) position of display buffer 112 and display buffer 132,display controller 104 can align display buffer 112 with display buffer132 according to the determined relative horizontal alignment offset andvertical alignment offset determined for display buffer 112 and displaybuffer 132. For example, with respect to configuration 202, displaycontroller 104 can align display buffer 112 to the right of displaybuffer 132 and adjust the vertical alignment according to the determinedvertical offset such that pixel row Y1 of display buffer 132 is alignedwith pixel row Y2 of display buffer 112. Thus, when the user moves thecursor (or other graphical object) from display device 130 (e.g.,display buffer 132) off the right edge of display device 130 at pixelrow Y1, the cursor will appear on display device 110 at the left edge ofdisplay device 110 (e.g., display buffer 112) at pixel row Y2. Thus, ifthe default alignment data correctly corresponds to the actual,physical, relative alignment of display device 110 and display device130, the cursor will appear in the appropriate (e.g., user expected)location when moved from one display device to another based on thelogical physical relationship of the displays, as observed by the user.

While the default alignment process is described above with reference totwo display devices, a similar process can be performed to determine therelative placement of display buffers corresponding to three or moredisplay devices to enable smooth transitions between multiple displaysin a multiple display system.

However, in some cases, the default alignment data of display device 110(e.g., display buffer 112) and display device 130 (e.g., display buffer132) may not represent the real-world alignment or relative positions ofthe two display devices. Thus, user device 102 may be configured tocause the user of user device 102 to engage in a display alignmentinteraction to determine the actual, real-world relative positions ofthe display devices in a multiple display system.

FIG. 3 is a diagram 300 illustrating a display buffer alignmenttechnique using a display alignment model. For example, the displayalignment model can be a two-dimensional or three-dimensional model thatrepresents the geometries and relative positions of the display devices(e.g., display device 110, display device 130) in space. As describedabove, user device 102 can have default alignment data associated witheach display device (e.g., display device 110, display device 130, etc.)connected to user device 102. The default alignment data can be used byuser device 102 to determine how to generate the display alignment modeland align the display buffers (e.g., display buffer 112, display buffer132, etc.) relative to each other in the model so that user device 102can present graphical objects (e.g., cursors, windows, etc.) that aremoved between display devices (e.g., between display buffers) at thelocations where the user would expect the graphical objects to appear onthe displays thereby providing smooth transition between displays. Forexample, the display alignment model can be used to align the displaybuffers for the display devices into a physically continuous and/orrepresentative mathematical space, as described further below.

To achieve this display alignment goal (e.g., a smooth transition,presentation of graphical objects at expected locations, etc.), displaycontroller 104 can generate the display alignment model using anormalized pixel transform buffer 302 (e.g., alignment buffer) todetermine how to move graphical objects from one display to another. Forexample, the normalized pixel transform buffer 302 can define a pixelspace that maps display buffers for the connected displays to pixellocations within the normalized pixel transform buffer 302 thatrepresent the real world spacing and/or relative orientations of theconnected displays as defined by the default alignment data describedabove or determined by the observed user interactions described below.

The pixel transform buffer 302 can include a pixel map (e.g.,two-dimensional array of pixels, three-dimensional array of pixels,etc.) that is sized large enough to include the display buffersassociated with each connected display and the relative spacing,orientations, positions, and/or alignments of the connected displays.The pixels of the pixel transform buffer 302 can be of a normalizedsize. For example, each of the displays connected to user device 102 mayhave different pixel sizes and/or pixel densities. Each of the pixels inthe displays, or display buffers, can be mapped to a pixel, or multiplepixels, in the pixel transform buffer 302 so that an appropriatetrajectory, curve, etc., can be determined for moving a graphical objectfrom one display buffer to another using the normalized, consistentpixel size array. Moreover, each pixel in the pixel transform buffer canbe mapped to real-world dimensions (e.g., lengths, sizes, distances,etc.) so that the display buffers can be appropriately spaced and thepixels within each display buffer can be mapped to appropriatecorresponding pixels in the pixel transform buffer 302. For example,pixel transform buffer 302 can be used to scale, rotate, translate,warp, or otherwise adjust the alignment of display buffers to accountfor differences in location, size, curvature, etc., of the variousdisplay devices (e.g., including keystone correction in projecteddisplays.

For example, if the alignment data (e.g., default or determined) fordisplay device 110 indicates that display device 110 is 3 inches abovethe surface upon which it is resting, then display buffer 112 can bespaced the pixel equivalent of 3 inches from the bottom of pixeltransform buffer 302 (e.g., pixel alignment buffer). Display buffer 112can be mapped to appropriate pixels within pixel transform buffer 302 sothat the size of display buffer 112 is appropriately represented. If thealignment data (e.g., default or determined) for display device 130indicates that display device 130 is 10 inches above display device 110,then display buffer 132 can be spaced the pixel equivalent of 10 inchesabove display buffer 112 corresponding to display device 110 in pixeltransform buffer 302. Display buffer 132 can be mapped to appropriatepixels within pixel transform buffer 302 so that the size of displaydevice 130 (e.g., and display buffer 132) is appropriately represented.The alignment data for display device 110 and/or display device 130 mayindicate that the horizontal spacing between display device 110 anddisplay device 130 is 5 inches. Thus, display buffer 112 and displaybuffer 132 can be placed the pixel equivalent of 5 inches apart in pixeltransform buffer 302. Thus, the current relative positions, alignmentand spacing of display devices 110 and 132 can be represented usingpixel transform buffer 302.

In some implementations, pixel alignment buffer 302 can be used todetermine how to move graphical objects from one display buffer (e.g.,and corresponding display device) to another display buffer. Forexample, the pixel transform buffer 302 can represent a two-dimensional(or three-dimensional) grid or graph that can be used to calculate thetrajectory (e.g., arc, curve, path, etc.) of graphical objects as theyare moved from one display buffer (e.g., display buffer 132) to anotherdisplay buffer (e.g., display buffer 112), determine errors in thecurrent relative alignments of the various display buffers correspondingto the displays connected to user device 102, and determine correctionsin display buffer alignments, as described below. Based on the relativedisplay buffer positions and alignments represented in the pixelalignment buffer 302, the trajectories, curves, and alignmentcorrections can be determined using not only the dimensions orspecifications of the represented display devices but also using, oraccounting for, the spacings (e.g., empty space) between display devicesand/or display buffers and relative display device positions andalignments as represented in pixel transform buffer 302.

FIG. 4 is a diagram 400 illustrating aligning display buffers based on aforced user interaction. For example, in response to detecting a newdisplay device 130 connected to user device 102 which is alreadyconnected to display device 110, display controller 104 can presentgraphical user interfaces on display devices 110 and 130 that prompt theuser to manipulate an alignment object on each display device toindicate the proper physical alignment of each display device. Thealignment data received through this user manipulation of the alignmentobjects can then be used to align the corresponding display buffers sothat the appropriate or expected user experience can be achieved whenmoving graphical objects between display devices.

Diagram 400 depicts and/or contrasts the alignment of physical displays(left side of diagram) and display buffers (right side of diagram) in asequence of stages 404-408 over time (indicated by arrow 402). Stage 404represents an example initial alignment of physical displays 110 and 130versus the alignment of the corresponding display buffers 112 and 132.The initial alignment can be determined and/or configured by user device102 when a new display device (e.g., display device 130) is detected,for example. The initial alignment can be determined and/or configuredby user device 102 based on a previous alignment or based on a defaultalignment of the display devices/display buffers. For example, userdevice 102 may receive user input initiating the display alignmentprocedure described below when the user rearranges the physical displaysand needs to realign the display buffers to match the new arrangement ofthe display devices.

Since at stage 404 the buffer alignment may be based on the defaultalignment data or previous (e.g., old) alignment data, the alignment ofdisplay buffers 112 and 132 may not correspond to the actual relativepositions and/or alignment of the physical display devices 110 and 130as configured by the user. For example, at stage 404, although thephysical display devices 110 and 130 are positioned with display device130 to the left of display device 110 and aligned vertically at theirbottom edges, corresponding display buffers 112 and 132 are positionedsuch that display buffer 132 (corresponding to display device 130) ispositioned to the right of display buffer 112 (corresponding to displaydevice 110) and centered vertically. Thus, because the display buffers112/132 do not have an alignment that matches the real-world alignmentof display devices 110/132, graphical objects that are moved between thedisplay devices 110/132 may not appear in the appropriate locations onthe display device to which the graphical objects are moved.

To allow the user to specify the alignment of the display devices110/130, and correspondingly, the display buffers 112/132, displaycontroller 104 may present graphical object 412 (e.g., alignment object)on display device 130 and graphical object 422 (e.g., alignment object)on display device 110. Since the data in display buffers 112/132determines what is displayed on each respective display device 110/132,to display the graphical objects on display device 110 and displaydevice 130, graphical object 422 must also be stored in display buffer112 and graphical object 412 must be stored in display buffer 132.

At stage 406, the user can provide input to align the alignment objectson display devices 110 and 130. For example, the user can provide input,and user device 102 can receive the input, to move the alignment object(e.g., primary alignment object) on the primary display device nearerthe alignment object (e.g., secondary alignment object) on the secondary(e.g., newly added) device until the primary alignment object is at theedge of the display device and aligned (e.g., vertically orhorizontally) with the secondary alignment object on the secondarydisplay device. While display controller 104 is moving the primaryalignment object in the direction specified by the received user input,display controller 104 can move the secondary alignment object in thedirection opposite of the user input so that each of the alignmentobjects move nearer to each other until aligned at the edge of theirrespective display devices.

As illustrated in the example of stage 406, when display device 110 isthe primary display device and display device 130 is the secondarydisplay device, the user can provide input to user device 102 to movegraphical object 422 presented on display device 110 toward graphicalobject 412 presented on display device 130. As graphical object 422 ismoved to the left toward display device 130 and graphical object 412,display controller 104 can move graphical object 412 to the right towarddisplay device 110 and graphical object 422. The user can continueproviding input to move graphical object 422 until graphical object 422is at the left edge of display device 110 and aligned vertically withgraphical object 412. Once aligned, the user can provide additional userinput (e.g., a mouse click, a button press, etc.) indicating thatgraphical object 422 is aligned with graphical object 412. Displaycontroller 104 can receive the user input indicating that the graphicalobjects are aligned, and determine the display location (e.g., pixellocation) of graphical object 422 on display device 110 and determinethe display location (e.g., pixel location) of graphical object 412 ondisplay device 130.

During the above alignment interaction of stage 406 using displaydevices 110 and 130, graphical objects 422 and 412 are being moved indisplay buffers 112 and 132. Since display buffers 112 and 132 are notyet aligned correctly, the movement of graphical objects 412 and 422 maybe used to realign the display buffers 112/132 to match the real-worldalignment of the display devices 110/132. For example, graphical object422 may move in display buffer 112 according to the alignment user inputand correspondingly too how graphical object 422 moves on display device110. However, since the display buffers 112/132 are not alignedcorrectly, graphical object 422 in display buffer 112 actually movesaway from graphical object 412 and display buffer 132. Based on thedirection of movement of graphical object 422 away from display buffer132, display controller 104 can determine that display buffer 112 ispositioned on the wrong side of display buffer 132 and rearrange therelative locations of the display buffers 112/132 such that displaybuffer 112 is positioned according to the direction of travel ofgraphical object 422. In the example of FIG. 4, display controller 104can reposition display buffer 112 to the right of display buffer 132, asrepresented by dashed arrow 430.

Once the relative arrangement (e.g., top, bottom, left, right) of thedisplay buffers is determined, display controller 104 can determine therelative alignment based on the final positions or locations ofgraphical objects 422 and 412 in display buffer 112 and 132,respectively. In the example, of FIG. 4, since display controller 104has determined that display buffer 112 should be positioned to the rightof display buffer 132 based on the user input, when the displaycontroller 104 receives the additional user input specifying the finalpositions of graphical objects 422 and 412, display controller 104 canuse the vertical positions (e.g., pixel row indexes) to align displaybuffers 112/132 vertically. If the display buffers were arrangedabove/below each other, then display controller 104 could use thehorizontal positions (e.g., pixel column indexes) of graphical objects422 and 412 to align display buffers 112/132 vertically. For example, atstage 408, display controller 104 can adjust the vertical position ofdisplay buffer 112 (or display buffer 132) such that the final locationsof graphical objects 422 and 412 are horizontally aligned. In theexample of stage 408, display controller 104 can move display buffer 112downward relative to display buffer 132 such that graphical objects 422and 412 are horizontally aligned. Thus, at the end of stage 408, thearrangement and/or alignment of display buffers 112/132 will correspondto, or match, the arrangement and/or alignment of the correspondingdisplay devices 110/130 in the real world.

FIG. 5 is a diagram 500 illustrating various relative displayalignments. While the forced user interaction alignment procedure wasdescribed above using an example a horizontal arrangement of displaysand vertical alignment adjustments, other display device arrangementsand alignments can be determined and corrected using the forced userinteraction procedure described above.

Diagram 500 includes display device/display buffer arrangement 510. Forexample, arrangement 510 is similar to the arrangement of FIG. 4.Arrangement 510 can include a primary display device corresponding todisplay buffer 514 and a secondary (e.g., added) display devicecorresponding to display buffer 512. Based on the user input movinggraphical objects 516 and 518, display controller 104 can determine thatsecondary display device was added to the left of the primary displaydevice and position display buffer 512 to the left of display buffer514. Moreover, based on the user input, display controller 104 candetermine the respective alignment locations (e.g., pixel rows) ofgraphical objects 516 and 518 and adjust the vertical position (e.g., upor down offset) of display buffer 512 relative to display buffer 514 sothat the alignment locations are aligned horizontally.

Diagram 500 includes display device/display buffer arrangement 520. Forexample, arrangement 520 is similar to the arrangement of FIG. 4.Arrangement 520 can include a primary display device corresponding todisplay buffer 522 and a secondary (e.g., added) display devicecorresponding to display buffer 524. Based on the user input movinggraphical objects 526 and 528, display controller 104 can determine thatsecondary display device was added to the right of the primary displaydevice and position display buffer 524 to the right of display buffer522. Moreover, based on the user input, display controller 104 candetermine the respective alignment locations (e.g., pixel rows) ofgraphical objects 526 and 528 and adjust the vertical position (e.g., upor down offset) of display buffer 524 relative to display buffer 522 sothat the alignment locations are aligned horizontally.

Diagram 500 includes display device/display buffer arrangement 530.Arrangement 530 can include a primary display device corresponding todisplay buffer 534 and a secondary (e.g., added) display devicecorresponding to display buffer 532. Based on the user input movinggraphical objects 536 and 538, display controller 104 can determine thatsecondary display device was added to above the primary display deviceand position display buffer 524 above display buffer 522. Moreover,based on the user input, display controller 104 can determine therespective alignment locations (e.g., pixel columns) of graphicalobjects 536 and 538 and adjust the horizontal position (e.g., right orleft offset) of display buffer 532 relative to display buffer 534 sothat the alignment locations are aligned vertically.

Diagram 500 includes display device/display buffer arrangement 540.Arrangement 540 can include a primary display device corresponding todisplay buffer 542 and a secondary (e.g., added) display devicecorresponding to display buffer 544. Based on the user input movinggraphical objects 546 and 548, display controller 104 can determine thatsecondary display device was added to below the primary display deviceand position display buffer 544 below display buffer 542. Moreover,based on the user input, display controller 104 can determine therespective alignment locations (e.g., pixel columns) of graphicalobjects 546 and 548 and adjust the horizontal position (e.g., right orleft offset) of display buffer 544 relative to display buffer 542 sothat the alignment locations are aligned vertically.

Accordingly, the forced user interaction alignment procedure describedabove can be implemented to align display buffers in a multiple displaysystem such that the arrangement (e.g., relative positions and/oralignment) of the display buffers closely matches or corresponds to thereal-world arrangement of the physical display devices. By arranging thedisplay buffers to correspond to the arrangement of the physical displaydevices, user device 102 and/or display controller 104 can presentgraphical objects on the display devices when moving the graphicalobject between display devices according to the user's expectations.Thus, user device 102 can be configured to avoid frustrating orconfusing the user and improve the user experience when working withmultiple display devices.

While FIG. 4 and FIG. 5 describe a forced interaction mechanism forreceiving user input specifying the relative positions and alignment ofdisplay devices in a multiple display system, display controller 104 mayobserve other user interactions with user device 102 and the multipledisplays connected to user device 102 to dynamically and/or covertlyadjust the alignment of display devices in a multiple display system toimprove the user experience. For example, these covert adjustments maybe performed based on historical user interaction data that describesmovements between display devices. The covert adjustments may beperformed based on real time or very recent user interaction data thatdescribes movements between display devices.

FIG. 6 illustrates an example graphical user interface 600 for provokingmultiple display interaction to determine display alignment. In someimplementations, GUI 600 can be presented when a changed displayarrangement is detected. For example, a change to the current displayarrangement can be detected when a new display device is connected touser device 102. For example, when user device 102, already connected todisplay device 110, detects a new display device 130 connected to userdevice 102, user device 102 can present GUI 600 across display device110 and display device 130. In particular, since display device 110 wasalready connected to user device 102, user device 102 may presentgraphical object 610 (e.g., a cursor, window, icon, etc.) on displaydevice 110. For example, user device 102 may present graphical object610 in the center, or near centered, on display device 110. To provoke across display user interaction, user device 102 can present a prompt 606on the newly connected display device 130. For example, prompt 606 mayrequire a user to provide some user input with respect to prompt 606(e.g., selection of the “ok” button 618) to dismiss prompt 606. Prompt606 can, for example, be a modal dialog box that simply requires theuser to provide input to acknowledge that a new display device has beenconnected to user device 102 and disallows user interaction with anyother graphical element until button 618 is selected.

In some implementations, a change to the current display arrangement canbe detected using various sensors. For example, a motion sensor attachedto a display device (e.g., a laptop computer, a tablet computer, asmartphone, etc.) can detect movement of the display device and causeGUI 600 to be presented. A sound sensor (e.g., microphone) of acomputing device can detect a sound associated with moving a displaydevice and cause GUI 600 to be presented. A light sensor can detect achange in ambient light emitted by the current display devices and cancause GUI 600 to be presented. For example, when user device 102,already connected to display device 110 and display device 130, detectschange to the current display configuration using one or more of thesensors described above, user device 102 can present GUI 600 acrossdisplay device 110 and display device 130. In particular, if user device102 is currently presenting graphical object 610 (e.g., a cursor,window, icon, etc.) on display device 110, user device 102 can present aprompt 606 on the display device 130 to provoke a cross display userinteraction. For example, prompt 606 may require a user to provide someuser input with respect to prompt 606 (e.g., selection of the “ok”button 618) to dismiss prompt 606. Prompt 606 can, for example, be amodal dialog box that simply requires the user to provide input toacknowledge that a new display device has been connected to user device102 and disallows user interaction with any other graphical elementuntil button 618 is selected.

In some implementations, the cross-display interaction can be analyzedto determine whether display device 110 and the new display device 130are arranged and/or aligned correctly. For example, display controller104 can analyze the trajectory and/or path (e.g., curve, arc, etc.)along which the user has moved graphical object 610 from display device110 to reach the destination location (e.g., “ok” button 618) on displaydevice 130. Based on the analysis, display controller 104 can determineif the user provided input to correct the path of graphical object 610when the graphical object was presented on display device 130. Forexample, humans typically plan movements involving coordination ofmultiple joints and degrees of freedom resulting in smooth andcontinuous arcs. If the user made a corrective input (e.g., indicated bya discontinuity of movement) when moving toward the destination locationon display device 130, display controller 104 can determine that thealignment or arrangement of the display buffer of display device 130with respect to the display buffer of display device 110 is incorrect.Display controller 104 can analyze the initial path of graphical object610 on display device 110 to determine how to make the arrangementand/or alignment correction necessary to properly align the displaybuffers, as described in further detail below.

FIG. 7 illustrates an example graphical user interface 700 for observingmultiple display interactions to determine display alignment. In someimplementations, display controller 104 can observe naturally occurring(e.g., spontaneous, unprovoked, etc.) user interactions with respect toGUI 700 to determine when display buffer alignment adjustments arerequired for a multiple display system. For example, user device 102 mayalready be connected to display device 110 and display device 130. Userdevice 102 can present (e.g., extend) GUI 700 across display device 110and display device 130. User device 102 may be presenting graphicalobject 710/720 (e.g., a cursor, window, icon, etc.) on display device110 or display device 130. Display controller 104 can determine (e.g.,based on a current user interaction, based on historical userinteractions) when a user interaction moves between display device 110and display device 130 and use the cross-display interactions todetermine when to correct the alignment of the corresponding displaybuffers.

In some implementations, the cross-display interaction can be analyzedto determine whether display device 110 and display device 130 arearranged and/or aligned correctly. For example, display controller 104can analyze the trajectory and/or path (e.g., curve, arc, etc.) alongwhich the user has moved graphical object 710 from display device 130 toreach the destination location (e.g., “window” menu item 714) on displaydevice 110. Similarly, display controller 104 can analyze the trajectoryand/or path (e.g., curve, arc, etc.) along which the user has movedgraphical object 720 from display device 110 to reach the destinationlocation (e.g., “Folder” icon 724) on display device 130. Based on theanalysis, display controller 104 can determine if the user providedinput to correct the path of graphical object 710/720 when the graphicalobject was presented on the destination display device. If the user madea corrective input when moving toward the destination location on thedestination display device, display controller 104 can determine thatthe alignment or arrangement of the display buffer of display device 130with respect to the display buffer of display device 110 is incorrect.In some implementations, display controller 104 can analyze the initialpath of graphical object 710/720 on the originating display device(e.g., display device 130 for graphical object 710, display device 110for graphical object 720) to determine how to make the arrangementand/or alignment correction necessary to properly align the displaybuffers of display device 110 and display device 130, as described infurther detail below.

Display controller 104 can perform this passive observation and displaybuffer alignment technique for each cross-display interaction. Displaycontroller 104 can perform this passive observation and display bufferalignment technique periodically on a sampling of historicalcross-display interactions. Display controller 104 can perform thispassive observation and display buffer alignment technique on a randomselection of current or recent cross display interactions.

FIG. 8 is a diagram 800 illustrating a cross-display movement pathindicative of misaligned display buffers. For example, diagram 800includes display device 802 and display device 804 that are part of amultiple display system. The user of the system may want to movegraphical object 810 (e.g., a cursor) presented on display device 804 toa destination or destination location 816 (e.g., target location) ondisplay device 816. To cause graphical object 810 to reach destinationlocation 816, the user can provide input to move graphical object 810along path 812. When the display buffers corresponding to display device802 and display device 804 are aligned correspondingly to the real-worldrelative positions (e.g., arrangement) of display device 802 and displaydevice 804, path 812 will typically follow a continuous curve from thestarting location of graphical object 810 across display device 804 andacross display device 802 to destination location 816.

However, when the display buffers corresponding to display device 802and display device 804 are not aligned correspondingly to the real-worldrelative positions (e.g., arrangement) of display device 802 and displaydevice 804, the user will need to provide input to correct the path 812of graphical object 810 to reach destination location 816, asillustrated by the discontinuity 814 in the curve of path 812. Forexample, because the display buffers were misaligned, graphical object810 entered the right edge of display device 802 at too high of alocation to continue along the curve corresponding to the path 812started on display device 804 and the user was required to make acorrection (e.g., discontinuity 814) to cause graphical object 810 toreach destination location 816.

In some implementations, display controller 104 can detect usercorrections to the movement path between display devices to determinewhen the display buffers for the display devices are misaligned. Forexample, display controller 104 can analyze path 812 to determine ifthere are any discontinuities introduced at the destination displaydevice 802 in the curve of the path started on display device 804. Whena discontinuity is found, or detected, display controller 104 can adjustthe alignment of the display buffers based on the curve of path 812started on display device 804, as described in further detail below.

FIG. 9 is a diagram 900 illustrating alignment of display buffers basedon observed cross-display interactions. For example, the cross-displayuser interactions can be provoked (as illustrated by FIG. 6) orpassively observed (as illustrated by FIG. 7). The analysis of the pathsof the cross-display interactions (e.g., user directed movements ofgraphical objects from device to device) and adjustments of displaybuffer alignments can be performed in response to detecting across-display interaction, periodically (e.g., nightly, weekly, etc.),in response to detecting specific contexts (e.g., user device is idle,not in use), or a combination thereof. For example, display controller104 can, on a nightly basis, filter historical user interaction data toidentify cross-display interactions occurring during the preceding day.Display controller 104 can analyze the paths of these cross-displayinteractions to determine whether any user corrections to the paths canbe detected. When a user correction is detected, display controller 104can adjust the relative positions, or arrangement, of the displaybuffers so that they more closely represent the arrangement of thephysical display devices.

Diagram 900 illustrates multiple stages 904, 906, 908 of procedure foraligning display buffers based on observed cross-display userinteractions. Each stage shows the physical alignment, or arrangement,of display devices 110 and 130 on the left side of timeline 902 and thevirtual alignment, or arrangement, of the corresponding display buffers112 and 132 on the right side of timeline 902.

The description of diagram 900 starts at stage 904 with misalignedbuffers 112 and 132 corresponding to display devices 110 and 130,respectively. For example, while the display device 130 is physicallyarranged to the left of display device 110, the display buffer 132corresponding to display device 130 is virtually arranged to the rightof display buffer 112 corresponding to display device 110. Moreover,while display devices 110 and 130 are aligned along their bottom edges,display buffers 112 and 132 are centered vertically relative to eachother. Thus, the misalignment of the display buffers compared to thephysical arrangement of the display devices will cause the user tocorrect the path of movement when moving graphical objects betweendisplay devices 110 and 130, as described in further detail below.

As the user moves graphical object 910 from display device 130 todestination 912 (e.g., target location) on display device 110, the userwill provide input to move graphical object 910 along a smooth path(e.g., arc, curve, trajectory, etc.) toward destination location 912.For example, the user can indicate destination location 912 by providingsome user input, such as mouse click, selection of a graphical object,releasing a graphical object (e.g., when moving a GUI object from onedisplay to another), or other movement termination event that indicatesdestination location 912. The process of analyzing a path and realigningdisplay buffers can be performed after receiving the input indicatingdestination location. For example, the analysis and realignment ofdisplay buffers can be performed after the user has confirmed that theuser intended to move between display devices, as implied or indicatedby the user input at the destination location 912 (e.g., the user'starget) on the second display device 110. In order to move graphicalobject 910 across display device 130, the location of graphical object910 in display buffer 132 must be updated based on the user input tomove graphical object 910 along path 914 so that display buffer 130 canbe used by display controller 104 to generate the frames of videopresented by display device 130.

In some implementations, display controller 104 can determine thedirection of travel of graphical object 910 along path 914 to determinethe relative position (e.g., to the right, left, above, below, etc.) ofdisplay device 110. For example, after display controller 104 receivesthe user input indicating destination location 912 on display device110, display controller 104 can analyze the direction of path 914 todetermine whether display buffer 112 for the destination display device110 is on the correct side of display buffer 132. As depicted at stage904, since display device 110 is physical arranged to the right ofdisplay 130, the user will provide input moving graphical object 910 tothe right on display device 130 to reach destination location 912.

However, since the display buffer 112 for display device 110 is arrangedto the left of display buffer 132, when graphical object 910 reaches theright edge of display device 130, the user will have to reverse courseand move graphical object 910 to the left off the left edge of displaydevice 130 to cause graphical object 910 to move onto display device110/display buffer 112. If this movement (e.g., the user hitting theedge of the display, moving in the opposite direction, and identifying adestination on another display device) is done within a period of timeor near continuously from start location to destination location,display controller 104 can identify this as a corrective movement withrespect to relative display positioning and rearrange the relativepositions of the display devices. For example, display controller 104can detect this corrective movement and determine that destinationdisplay device (e.g., display device 112) is actually located in theinitial direction of travel (e.g., to the right) of path 914 on displaydevice 132. Display controller 104 can then rearrange the relativepositions of display buffer 112 and display buffer 132 so that displaybuffer 112 is to the right of display buffer 132, as depicted in stage906. Thus, when the user subsequently provides input to move graphicalobject 910 to the right from display device 130 to display device 110,the graphical object 910 will move to the right off display device 130and onto display device 110 as expected given the physical orientationof the display devices. Graphical object 910 will show up as expected ondisplay device 110 because the display buffers 112 and 132 are nowarranged similar to the relative positions of display device 110 anddisplay device 130.

However, even though the display buffers 112/132 may be arranged on theappropriate sides of each other, the display buffers 112/132 may not bealigned correctly to produce the expected behavior when graphical object910 is moved from display device 130 to display device 110 (or viceversa).

Referring to stage 906, a user may provide input to user device 102 tomove graphical object 910 from display device 130 to destinationlocation 912 on display device 110. When initiating the move on displaydevice 130, the user will typically move graphical object 910 along apath 916 (e.g., a continuous curve) on display device 130 towarddestination location 912. However, when the display buffers for displaydevice 130 and display device 110 (e.g., display buffer 132 and displaybuffer 112, respectively) are misaligned, in this case verticallymisaligned, with respect to the relative positions of display device 130and display device 110, the user will need to provide input correctingthe path of graphical object 910 on display device 110 so that graphicalobject 910 reaches destination location 912 on display device 912. Forexample, when display buffer 132 and 112 are misaligned with respect tothe relative physical positions of display device 130 and display device110, graphical object 910 will appear too low (or too high) on displaydevice 110 (and display buffer 112). Thus, the user will need to movegraphical object 910 in an upwardly curved path 918 to cause graphicalobject 910 to reach destination location 912 on display device 110. Thechange in trajectory of the initial path 916 creates a discontinuity inthe curve of path 916 from display device 130 to destination location912 on display device 110 that can be used by user device 102 todetermine when display buffer 132 and display buffer 112 are misaligned.For example, use device 102 can fit a curve to path 920 (e.g., a curvebased on the combination of path 916 and path 918) between displaybuffer 132 and display buffer 112 (e.g., display device 130 and displaydevice 110, respectively) using well-known curve fitting algorithms.User device 102 can numerically integrate the curve of path 920 anddetermine if there are any discontinuities. For example, user device 102can determine if there are any discontinuities in the first derivativeor the second derivative of the curve. If a discontinuity in the curveof path 920 is identified, then user device 102 can adjust the relativealignment of display buffer 132 and display buffer 112.

At stage 908, user device 102 can adjust the relative alignment ofdisplay buffer 132 and display buffer 112 based on the curve of initialpath 916 on display buffer 132 (e.g., display device 130). For example,user device 102 can use the display alignment model of FIG. 3 to projectthe curve of initial path 916 beyond the edge of display buffer 132 inthe direction of the movement of graphical object 910 (e.g., towardsdisplay buffer 912). User device 102 can then adjust the alignment ofdisplay buffer 112 (e.g., indicated by dashed downward arrow and dashedoutline of previous location of display buffer 112) so that thedestination location 912 indicated by the user is on the projected curveof path 916. Thus, display buffer 112 can be aligned with display buffer132 so that the alignment of the display buffers more closely matchesthe alignment of the physical display devices 130 and 110.

In some implementations, user device 102 can adjust the alignment ofdisplay buffers based on multiple user interactions with the displaydevice 110 and display device 130. For example, a single multi-displayuser interaction may not provide enough information to correctly aligndisplay buffers 112 and 132 to match the physical alignment of displaydevices 110 and 130. When display devices 110 and 130 are positionedleft and right of each other, as depicted in FIG. 9, user device 102 mayadjust the relative alignment of display buffer 112 downward withrespect to display buffer 132 based on a user interaction but thisadjustment may not account for the spacing between display devices.Thus, on a subsequent interaction the user may provide input to correctthe input path again when moving between displays because user device102 did not calculate where to display the graphical object 910 based ona correct spacing between display buffer 112 and display device 132 eventhough the vertical alignment was previously adjusted. By adjusting therelative alignment of display buffer 112 to fit multiple userinteractions (e.g. multiple path curves and corresponding destinationlocations), user device 102 can adjust the alignment of display buffer112 up and down (e.g., or left and right if displays are arrangedvertically) and the spacing between display buffers 112 and 132 so thatfor all of the analyzed user interactions, all of the destinationlocations intersect the corresponding path curves. Thus, adjusting thealignment of the display buffers based on multiple user interactions(e.g., historical user interactions, previous user interactions) mayresult in a more accurate alignment of the display buffers that moreclosely represents the actual physical alignment of display devices 110and 130.

FIG. 10 is a diagram 1000 illustrating alignment of multiple externaldisplay buffers based on observed cross-display interactions. Forexample, diagram 1000 illustrates how a computing device can alignexternal display buffers for multiple external display devices when theexternal display devices are adjacent. The alignment process illustratedby diagram 1000 can be performed by provoking a user interaction (e.g.,presenting GUI targets that the user must select) or by otherwiseobserving user interactions with graphical object (e.g., GUI targets) ondifferent external displays, as described herein.

In some implementations, user device 102 can detect the addition of oneor more external display devices 130 and/or 1010 to user device 102 thatincludes display device 110. User device 102 can detect an event thatmay indicate a change in the alignment, orientation, or configuration ofthe various display devices 130, 1010, and/or 110 with respect to eachother. In response to detecting a change in the configuration of thedisplay devices 130, 1010, and/or 110 connected to user device 102, userdevice 102 can initiate a provoked (e.g., forced, prompted, etc.) orobserved alignment process, as described herein.

At stage 1004, user device 102 can initially align display buffers 1012,132, and or 112 corresponding to physical display devices 130, 1010, and110 according to default alignment data or previously determinedalignment data, as described above. In the example of stage 1004, userdevice 102 can initially align the display buffers with display buffer1012 on the far left, display buffer 112 on the far right, and displaybuffer 132 between display buffer 1012 and display buffer 112. Asillustrated by the physical display alignment of display device 130,display device 1010, and display device 110, the initial display bufferalignment may not represent the real-world physical locations of thedisplay devices. For example, the real-world alignment of the displaydevices may have display device 130 positioned at the far left, displaydevice 110 at the far right, and display device 1010 between (e.g., inthe middle of) display device 110 and display device 130.

To align or position the display buffers so that they match thereal-world alignment of the physical display devices and will thereforepresent graphical objects in the appropriate locations on the displaydevices when the graphical objects are moved between display devicesand/or display buffers, user device 102 can perform a display bufferalignment operation based on observed user input (e.g., provoked orpassive), as described herein above and below.

When performing the provoked or prompted display buffer alignmentoperation, user device 102 can, in response to detecting a change indisplay device alignment (e.g., a repositioning or display deviceaddition), present graphical object 1020 (e.g., a cursor, a pointer, aninput position indicator, etc.) on one of the display devices (e.g.,display device 110) and present graphical targets 1022, 1024 (e.g.,prompts that require user input, prompt 606, etc.) on the other displaydevices (e.g., display device 130, display device 110. These graphicalobjects will, of course, be correspondingly represented in displaybuffers 1012, 132, and 112.

In some implementations, graphical object 1020 can be initiallypresented on the primary display device (e.g., display device 110) whenperforming the alignment operation. In some implementations, graphicalobject 1020 can be presented at its current location on a previouslyconnected display device. For example, if user device 102 was previouslyconnected to display device 110 and display device 130 and graphicalobject 1020 was located on display device 110 when the alignmentoperation was initiated by user device 102, then graphical object 1020can remain presented at its location on display device 110.

In some implementations, graphical targets 1022, 1024 (e.g., prompts,passive targets, etc.) can be presented on the other display devices(e.g., display devices 130, 1010, etc.) that are not currentlypresenting graphical object 1020. For example, each of the displaydevices 130, 1010 can present one prompt that requests user input.Graphical targets 1022, 1024 can be presented at different locations oneach display device so that the user will move graphical object 1020along different trajectories to reach each target. For example,graphical target 1022 can be presented at a different vertical location(e.g., near the top edge of the screen) on display device 130 than thevertical location (e.g., near the bottom edge of the screen) at whichgraphical target 1024 is presented on display device 1010. Graphicaltarget 1022 can be presented at a different horizontal location (e.g.,near the left edge of the screen) on display device 130 than thevertical location (e.g., near the right edge of the screen) at whichgraphical target 1024 is presented on display device 1010.

When graphical targets 1022, 1024 are presented on display device 120and display device 1010, the user may provide input to user device 102to move graphical object 1020 toward one of the graphical targets 1022,1024. Since display device 1010 is physically positioned closer todisplay device 110 than display device 130, the user will likely movegraphical object 1020 from its location on display 110 toward GUI target1024 on display device 1010. When graphical object 1020 is moved off theedge (e.g., left edge) of the screen of display device 110, user device102 can present graphical object 1020 on the right edge of the screenson display device 130 and display device 1010. For example, since theuser has moved graphical object 1020 to the left off display device 110,user device 102 can initially determine that display device 130 and/ordisplay device 1010 is positioned to the left of display device 110 andpresent graphical object 1020 on the right side of the screens ofdisplay device 130 and display device 1010. To be clear, graphicalobject 1020 will now be presented on both display device 130 and displaydevice 1010.

As the user moves graphical object 1020 on display device 130 anddisplay device 1010 (e.g., the same user input moves graphical object1020 on both devices), user device 102 can record the trajectory of themovement until the user moves graphical object 1020 to one of thegraphical targets 1022 or 1024. For example, since graphical targets1022 and 1024 are positioned in horizontally and/or vertically differentlocations on each display device 130, 1010, the user input movinggraphical object 1020 on both devices will not cause graphical object1020 to intersect both graphical target 1022 and graphical target 1024at the same time. Thus, when the user moves graphical object 1020 to agraphical target 1022 or 1024 and provides input selecting the graphicaltarget (e.g., an ‘ok’ button, a file, a folder, etc.), user device 102can determine which display device the user is focused on and align thatdisplay device with the display device that initially presentedgraphical object 1020. For example, since the location of graphicalobject 1020 corresponds to the location of graphical target 1024 ondisplay device 1010 when the user input indicating a selection wasreceived, user device 102 can determine that the user's focus was ondisplay device 1010. Conversely, since the location of graphical object1020 corresponds to a location where no graphical target is located (asillustrated by empty circle 1026) on display device 130 when the userinput indicating a selection was received, user device 102 can determinethat the user's focus was not on display device 1010.

As illustrated in the example of stage 1004, the user may move graphicalobject 1020 off the left edge of display device 110 and select graphicaltarget 1024 on display device 1010 since display device 1010 isphysically positioned closest to display device 110. After receiving theuser input selecting graphical target 1024, user device 102 can analyzethe path (e.g., trajectory) along which the user moved graphical object1020 to reach graphical target 1024 (e.g., destination location 1024),to determine how to align display buffer 1011 with display buffer 112,as described above. In this example, since the display buffer 1012 ispositioned incorrectly with respect to display buffer 112 and displaybuffer 132, user device 102 can reposition display buffer 1012 so thatit is adjacent to display buffer 112 and between display buffer 112 anddisplay buffer 132, as illustrated by the arrow between stage 1004 andstage 1006.

At stage 1006, user device 102 has aligned display buffer 1012 withdisplay buffer 112 but user device 102 has not yet aligned displaybuffer 132 with respect to display buffer 1012 and/or display buffer112. Thus, the display buffer alignment operation can continue at stage1006 with graphical object 1020 presented on display device 1010 at thelocation where graphical target 1024 was previously presented. Forexample, graphical target 1024 can be dismissed or hidden by user device102 after being selected by the user. Since graphical target 1022 wasnot selected by the user, user device 102 can continue to presentgraphical target 1022 on display device 130. Seeing the graphical target1022 (e.g., a user prompt), the user may move graphical object 1020 ontodisplay device 130 to select graphical target 1022. After receiving userinput to move graphical object 1020 to the location of graphical target1022 and receiving user input selecting graphical target 1022, userdevice 102 can align display buffer 132 with respect to display buffer1012 and/or display buffer 112 based on the path or trajectory alongwhich the user moved graphical object 1020 to reach graphical target1022 (e.g., the destination location), as described above. Thus, displaybuffers 112, 132, and 1012 can be aligned by observing how the usermoves graphical object 1020 between display devices to select thegraphical targets.

While the above description of FIG. 10 primarily describes analyzing aprompted (e.g., provoked) user interaction to align display buffers, asimilar approach can be used to align display buffers during anunprompted user interaction. For example, instead of presenting userinput prompting graphical targets 1022, 1024, user device 102 canmonitor user input that moves graphical object 1020 to passive graphicaltargets 1022, 1024. For example, passive graphical targets cancorrespond to menus, files, folders, or other graphical objects that arenormally presented on a display screen during normal operation. Userdevice 102 can monitor the user's movement of graphical object 1020,receive a user selection input selecting a graphical target, determinewhich display (e.g., display 1010, display 130) is currently presentinga graphical target (e.g., graphical target 1024) at the location wherethe user selection input was received, and align the correspondingdisplay buffer based on the path along which graphical object 1020 wasmoved by the user. Thus, display buffers 112, 132, and 1012 can bealigned by observing how the user moves graphical object 1020 betweendisplay devices to select the non-prompting graphical targets.

FIG. 11 is a diagram 1100 illustrating alignment of multiple externaldisplay buffers based on observed cross-display interactions. Forexample, diagram 1100 illustrates how a computing device can alignexternal display buffers for multiple external display devices when theexternal display devices are not adjacent. The alignment processillustrated by diagram 1100 can be performed by provoking a userinteraction (e.g., presenting GUI targets that the user must select) orby otherwise observing user interactions with graphical object (e.g.,targets) on different external displays, as described herein.

In some implementations, user device 102 can detect the addition of oneor more external display devices 130 and/or 1010 to user device 102 thatincludes display device 110. User device 102 can detect an event thatmay indicate a change in the alignment, orientation, or configuration ofthe various display devices 130, 1010, and/or 110 with respect to eachother. In response to detecting a change in the configuration of thedisplay devices 130, 1010, and/or 110 connected to user device 102, userdevice 102 can initiate an observed (e.g., provoked or unprovoked)alignment process, as described herein.

At stage 1104, user device 102 can initially align display buffers 1012,132, and or 112 corresponding to physical display devices 130, 1010, and110 according to default alignment data or previously determinedalignment data, as described above. In the example of stage 1104, userdevice 102 can initially align the display buffers with display buffer1012 on the far left, display buffer 112 on the far right, and displaybuffer 132 between display buffer 1012 and display buffer 112. Asillustrated by the physical display alignment of display device 130,display device 1010, and display device 110, the initial display bufferalignment may not represent the real-world physical locations of thedisplay devices. For example, the real-world alignment of the displaydevices may have display device 1010 positioned at the far left, displaydevice 130 at the far right, and display device 110 between (e.g., inthe middle of) display device 130 and display device 1010.

To align or position the display buffers so that they match thereal-world alignment of the physical display devices and will thereforepresent graphical objects in the appropriate locations on the displaydevices when the graphical objects are moved between display devicesand/or display buffers, user device 102 can perform a display bufferalignment operation based on observed user input (e.g., provoked orpassive), as described herein above and below.

When performing the provoked or prompted display buffer alignmentoperation, user device 102 can, in response to detecting a change indisplay device alignment (e.g., a repositioning or display deviceaddition), present graphical object 1120 (e.g., a cursor, a pointer, aninput position indicator, etc.) on one of the display devices (e.g.,display device 110) and present graphical targets 1122, 1124 (e.g.,prompts that require user input, prompt 606, etc.) on the other displaydevices (e.g., display device 130, display device 110. These graphicalobjects will, of course, be correspondingly represented in displaybuffers 1012, 132, and 112 from which the graphics presented on thedisplays are obtained and/or generated.

In some implementations, graphical object 1120 can be initiallypresented on the primary display device (e.g., display device 110) whenperforming the alignment operation. For example, in a multiple displaysystem, one of the display devices can be identified as a primarydisplay device. In some implementations, graphical object 1120 can bepresented at its current location on a previously connected displaydevice. For example, if user device 102 was previously connected todisplay device 110 and display device 130 and graphical object 1120 waslocated on display device 110 when the alignment operation was initiatedby user device 102, then graphical object 1120 can remain presented atits location on display device 110.

In some implementations, graphical targets 1122, 1124 (e.g., prompts,passive targets, etc.) can be presented on the other display devices(e.g., display devices 130, 1010, etc.) that are not currentlypresenting graphical object 1120. For example, each of the displaydevices 130, 1010 can present one prompt that requests user input.Graphical targets 1122, 1124 can be presented at different locations oneach display device to induce the user to move graphical object 1120along different trajectories to reach each target. For example,graphical target 1122 can be presented at a different vertical location(e.g., near the top edge of the screen) on display device 130 than thevertical location (e.g., near the bottom edge of the screen) at whichgraphical target 1124 is presented on display device 1010. Graphicaltarget 1122 can be presented at a different horizontal location (e.g.,near the left edge of the screen) on display device 130 than thevertical location (e.g., near the right edge of the screen) at whichgraphical target 1124 is presented on display device 1110.

When graphical targets 1122, 1124 are presented on display device 120and display device 1010, the user may provide input to user device 102to move graphical object 1120 toward one of the graphical targets 1122,1124. Since display device 1010 and display device 130 are bothphysically located near display device 110, the user may move graphicalobject 1120 from its location on display 110 toward GUI target 1124 ondisplay device 1010 or toward GUI target 1122 on display device 130. Inthe example of FIG. 11, the user has chosen to move graphical object1120 toward graphical target 1124 presented by display device 1010. Whengraphical object 1120 is moved off the edge (e.g., left edge) of thescreen of display device 110, user device 102 can present graphicalobject 1120 on the right edge of the screens on display device 130 anddisplay device 1010. For example, since the user has moved graphicalobject 1120 to the left off display device 110, user device 102 caninitially determine that display device 130 and/or display device 1010is positioned to the left of display device 110 and present graphicalobject 1120 on the right side of the screens of display device 130 anddisplay device 1010. To be clear, graphical object 1020 will now bepresented on both display device 130 and display device 1010.

As the user moves graphical object 1120 on display device 130 anddisplay device 1010 (e.g., the same user input moves graphical object1120 on both display devices), user device 102 can record the trajectoryof the movement until the user moves graphical object 1120 to one of thegraphical targets 1122 or 1124. For example, since graphical targets1122 and 1124 are positioned in horizontally and/or vertically differentlocations on each display device 130, 1010, the user input movinggraphical object 1120 on both devices will not cause graphical object1020 to intersect both graphical target 1122 and graphical target 1124at the same time. Thus, when the user moves graphical object 1120 to agraphical target 1122 or 1124 and provides input selecting the graphicaltarget (e.g., an ‘ok’ button, a file, a folder, etc.), user device 102can determine which display device the user is focused on and align thatdisplay device with the display device that initially presentedgraphical object 1120. For example, since the location of graphicalobject 1120 corresponds to the location of graphical target 1124 ondisplay device 1010 when the user input indicating a selection wasreceived, user device 102 can determine that the user's focus was ondisplay device 1010. Conversely, since the location of graphical object1120 corresponds to a location where no graphical target is located (asillustrated by empty circle 1126) on display device 130 when the userinput indicating a selection was received, user device 102 can determinethat the user's focus was not on display device 1010.

As illustrated in the example of stage 1104, the user may move graphicalobject 1120 off the left edge of display device 110 and select graphicaltarget 1124 on display device 1010 since display device 1010. Afterreceiving the user input selecting graphical target 1124, user device102 can analyze the path (e.g., trajectory) along which the user movedgraphical object 1120 to reach graphical target 1124 (e.g., destinationlocation 1124), to determine how to align display buffer 1012 withdisplay buffer 112, as described above. In this example, since thedisplay buffer 1012 is positioned incorrectly with respect to displaybuffer 112 and display buffer 132, user device 102 can repositiondisplay buffer 1012 so that it is next to display buffer 112 and betweendisplay buffer 112 and display buffer 132, as illustrated by the arrowbetween stage 1104 and stage 1106 indicating the repositioning ofdisplay buffer 1012.

At stage 1106, user device 102 has aligned display buffer 1012 withdisplay buffer 112 but user device 102 has not yet aligned displaybuffer 132 with respect to display buffer 1012 and/or display buffer112. Thus, the display buffer alignment operation can continue at stage1106 with graphical object 1120 presented on display device 1010 at thelocation where graphical target 1124 was previously presented. Forexample, graphical target 1124 can be dismissed or hidden by user device102 after being selected by the user. Since graphical target 1122 wasnot selected by the user, user device 102 can continue to presentgraphical target 1122 on display device 130. Seeing the graphical target1122 (e.g., a user prompt), the user may move graphical object 1120 todisplay device 130 through display device 110 to select graphical target1022 as indicated by path 1130.

After receiving user input to move graphical object 1120 to the locationof graphical target 1122 and/or receiving user input selecting graphicaltarget 1122, user device 102 can align display buffer 132 with respectto display buffer 1012 and/or display buffer 112 based on the path ortrajectory (e.g., path 1130) along which the user moved graphical object1120 to reach graphical target 1122 (e.g., the destination location), asdescribed above. Thus, display buffers 112, 132, and 1012 can be alignedby observing how the user moves graphical object 1120 between displaydevices to select the graphical targets.

While the above description of FIG. 11 primarily describes analyzing aprovoked (e.g., prompted) user interaction to align display buffers, asimilar approach can be used to align display buffers during anunprompted user interaction. For example, instead of presenting userinput prompting graphical targets 1122, 1124, user device 102 canmonitor user input that moves graphical object 1120 to passive graphicaltargets 1122, 1124. For example, passive graphical targets cancorrespond to menus, files, folders, or other graphical objects that arenormally presented on a display screen during normal operation. Userdevice 102 can monitor the user's movement of graphical object 1120,receive a user selection input selecting a graphical target, determinewhich display (e.g., display 1010, display 130) is currently presentinga graphical target (e.g., graphical target 1124) at the location wherethe user selection input was received, and align the correspondingdisplay buffer based on the path along which graphical object 1120 wasmoved by the user. Thus, display buffers 112, 132, and 1012 can bealigned by observing how the user moves graphical object 1020 betweendisplay devices to select the non-prompting graphical targets.

Example Processes

To enable the reader to obtain a clear understanding of thetechnological concepts described herein, the following processesdescribe specific steps performed in a specific order. However, one ormore of the steps of a particular process may be rearranged and/oromitted while remaining within the contemplated scope of the technologydisclosed herein. Moreover, different processes, and/or steps thereof,may be combined, recombined, rearranged, omitted, and/or executed inparallel to create different process flows that are also within thecontemplated scope of the technology disclosed herein. Additionally,while the processes below may omit or briefly summarize some of thedetails of the technologies disclosed herein for clarity, the detailsdescribed in the paragraphs above may be combined with the process stepsdescribed below to get a more complete and comprehensive understandingof these processes and the technologies disclosed herein.

FIG. 12 is flow diagram of an example process 1200 for aligning displaybuffers based on a forced user interaction. For example, process 1200can be performed by user device 102 in response to detecting a newconnection to a new display device.

At step 1202, user device 102, already connected to a first displaydevice, can establish a connection to a second display device. Forexample, user device 102 can be a laptop computer with a built-indisplay that establishes a connection to another display through a videocable. User device 102 can be a desktop computer connected to anexternal display that establishes a connected to another externaldisplay through a video cable. User device 102 can be tablet computerwith a built-in display that establishes a wireless connection to a wallmounted display (e.g., television, monitor, etc.).

At step 1204, user device 102 can present a first graphical object thefirst display device and a second graphical object on the second displaydevice. For example, display controller 104 can present a firstgraphical object (e.g., a box, a square, an arrow, etc.) on the firstdisplay device and a second graphical object (e.g., a box, a square, anarrow, etc.) on the second display device. To do so, display controller104 can store data representing the first graphical object and thesecond graphical object in the display buffers for the respective firstand second display devices. Frame data can be generated based on thedata in the display buffers and the frames can be sent to the displaydevices according to the respective frame rates of the display devices,according to well-known practices.

At step 1206, user device 102 can receive a first user input for movingthe first graphical object on the first display device and the secondgraphical object on the second display device. For example, displaycontroller 104 can prompt the user to move the first graphical object onthe first display device toward the second graphical object on thesecond display device to align the first and second graphical objects atthe edge of their respective display devices. The user can provide inputto move the first and second graphical objects using various inputdevices, such as a mouse, a trackpad, a keyboard, etc.

At step 1208, user device 102 can, in response to receiving the firstuser input, move the first graphical object on the first display deviceand the second graphical object on the second display device. Forexample, display controller 104 can move the first and second graphicalobjects in the display buffers corresponding to the first and seconddisplay devices to cause the first and second graphical objects toappear to move across the first and second display devices according tothe receives first user input. For example, if the user provides inputto move the first graphical object in one direction (e.g., upward and tothe right on the first display device), display controller 104 can movethe second graphical object in the opposite direction (e.g., downwardand to the left on the second display device) so that at some point inthe movement the first and second graphical objects may be aligned.

At step 1210, user device 102 can receive a second user input indicatingthat a first current location of the first graphical object on the firstdisplay device is aligned with a second current location of the secondgraphical object on the second display device. For example, displaycontroller 104 can receive a second user input (e.g., a mouse click, aparticular keyboard input, etc.) indicating that the first and secondgraphical objects are aligned. For example, the second user input (e.g.,a mouse click) can be different than the first user input (e.g., mousemovement) moving the first and second graphical objects. The user mayprovide the second user input when the first and second graphicalobjects are horizontally or vertically aligned to indicate that thecurrent locations of the first graphical object and the second graphicalobjects are aligned in the current physical orientation of the displaydevices. For example, when the first display device and the seconddisplay device are positioned to the left and right of each other, thenthe user may provide the second user input when the first and secondgraphical objects are horizontally aligned (e.g., the row of pixelswhere the first graphical object is located is aligned with the row ofpixels where the second graphical object is located). When the firstdisplay device and the second display device are positioned above andbelow each other, then the user may provide the second user input whenthe first and second graphical objects are vertically aligned (e.g., thecolumn of pixels where the first graphical object is located is alignedwith the column of pixels where the second graphical object is located).

At step 1212, user device 102 can, in response to receiving the seconduser input, align a first display buffer of the first display devicewith a second display buffer of the second display device based on thefirst current location and the second current location. For example,display controller 104 can align the display buffer corresponding to thefirst display device with the display buffer corresponding to the seconddisplay device based on the alignment locations on the display devices(e.g., and in the respective display buffers) indicated by the user withthe second user input. The alignment can be performed by determining anoffset value corresponding to the number of pixel rows or columnsbetween the alignment positions on the respective display devices orwithin the corresponding display buffers. For example, the number ofpixel rows/columns can be determined by calculating the difference inthe pixel row/column indices for each alignment location. For example,when the physical display devices (e.g., and display buffers) arepositioned above and below each other, the user indicated alignmentlocation for the first graphical object may be at pixel column 20 andthe user indicated alignment location for the second graphical objectpixel column 75. Thus, the column offset value for aligning the displaybuffers can be −50 when moving from the second display buffer to thefirst display buffer and +50 when moving from the first display bufferto the second display buffer. By adding or subtracting the column offsetvalue to the last location of the graphical object in the previousdisplay buffer of a graphical object being moved between display buffers(e.g., display buffers), display controller 104 can cause the graphicalobject to appear in the expected location on the destination displaydevice when moving the graphical object between display devices. Similarrow offset values can be calculated to align display buffers and movegraphical objects between display buffers (e.g., display devices) whenthe display devices (e.g., display buffers) are arranged to the rightand left each other.

FIG. 13 is a flow diagram of an example process 1300 for aligningdisplay buffers based on observed interactions. For example, process1300 can be performed by a display controller 104 on user device 102 toautomatically detect and correct alignment issues between displaydevices (e.g., display buffers) in a multiple display system. Process1300 can be implemented to covertly or overtly refine the alignment ofdisplay buffers so that the display buffer alignment more closelymatches the alignment or positioning of the corresponding physicaldisplay devices in the real world. When the display buffers more closelymatch the physical alignment of the display devices, graphical objectsmoved between display devices are more likely to appear in user expectedlocations on the display devices thereby improving the user experience.

At step 1302, user device 102 can detect movement of a graphical objectfrom a first display buffer corresponding to a first display device to asecond display buffer corresponding to a second display device. Forexample, as a user provides input to move a graphical object (e.g.,cursor, folder, window, etc.) from a first display device connected touser device 102 to a second display device connected to user device 102,display controller 104 of user device 102 can update the location of thegraphical object in the display buffers corresponding to respectivedisplay devices. When display controller 104 detects movement of thegraphical object between display devices (e.g., from the first displaybuffer to the second display buffer, or from the second display bufferto the first display buffer), display controller 104 can analyze themovement to determine if an adjustment to the alignment of the displaybuffers is required.

At step 1304, user device 102 can determine a path corresponding to themovement of the graphical object from a first location in the firstdisplay buffer to a second location in the second display buffer. Forexample, display controller 104 can determine a path along which theuser has moved the graphical object from the first location in the firstdisplay buffer (e.g., on the first display device) to a second location(e.g., destination location) in the second display buffer (e.g., on thesecond display device). The second location, or destination location,can be identified by some path terminating action or input taken by theuser. For example, the destination location for the path can beidentified by user input selecting a graphical object, menu, or someother user input indicative of the target of the movement of thegraphical object.

In some implementations, user device 102 can exclude paths, or portionsof paths, that are not appropriate or valid for use when determining howto align display buffers. For example, if the path on the initialdisplay is wandering (not well described as an arc) then the userpossibly lacked intention and the path is consequently not predictive ofan alignment issue, nor useful for making an alignment correction.However, if the initial path is not a ballistic arc, it might be anintentional line and then probably very predictive of the intended pathto the target and any deviation on the new display is similarlyindicative of an alignment error between display buffers.

At step 1106, user device 102 can detect a correction in the pathassociated with the second display buffer. For example, displaycontroller 104 can fit a curve to the path of the graphical object as itmoves between display buffers and determine whether a discontinuity(e.g., correction) exists in the curve of the path. For example, whendisplay controller 104 detects a first or second derivativediscontinuity in the curve of the path, display controller 104 candetermine that the user made a correction to the path of the graphicalobject as the user moved the graphical object from the first displaybuffer to the second display buffer. For example, the discontinuity maybe found in the portion of the path corresponding to, or moving across,the second display buffer when the relative positions of the displaybuffers (e.g., right/left, top/bottom) are correct but the displaybuffers are misaligned horizontally or vertically. The discontinuity maybe found in the portion of the path corresponding to, or moving across,the first display buffer when the relative positions of the displaybuffers are incorrect (e.g., right/left positions, top/bottom positions,etc.)

At step 1308, user device 102 can adjust a position of the seconddisplay buffer relative to the first display buffer based on thedetected correction. For example, in response to detecting a correctionin the path of the graphical object, display controller 104 can move theposition of the second display buffer relative to the first displaybuffer such that the destination location intersects a projection of theinitial path of the graphical object (e.g., the portion of the pathbefore the user correction). Thus, after adjustment, if the user hadcontinued to move the graphical object along the initial path (e.g., acontinuous curve extended or projected from the initial path), thegraphical object would have reached the destination location withoutuser input correcting the path.

The above process 1300 describes an display buffer alignment processthat adjusts the alignment of a second display buffer relative to afirst display buffer by moving the entire second display buffer suchthat a destination location intersects a projected or extended curve ofan initial portion of a path from a starting location to the destinationlocation. However, process 1300 can be performed using multiple userinput path/destination location data sets to more accurately align thedisplay buffers. By aligning the second display buffer such thatmultiple destination locations intersect multiple correspondingprojected user input paths, display controller 104 can more accuratelydetermine not only alignment offsets (e.g., horizontal or verticalalignment), but also spacing between display devices so that displaycontroller 104 can better determine where to location graphical objectsin display buffers and on display devices when moving graphical objectsbetween display buffers and display devices.

Additionally, the above display alignment approaches could be used togenerate different display alignments for different users. For example,by using the forced user interaction approach and/or the observed userinteraction approaches described above, the computing device cangenerate different display alignments for different users that use thesame computer and display devices. This may be beneficial when the usershave different physical characteristics (e.g., right vs. left handusers, short vs. tall users, etc.) and/or when the users positionthemselves differently with respect to the display devices when usingthe computer.

Graphical User Interfaces

This disclosure above describes various Graphical User Interfaces (GUIs)for implementing various features, processes or workflows. These GUIscan be presented on a variety of electronic devices including but notlimited to laptop computers, desktop computers, computer terminals,television systems, tablet computers, e-book readers and smart phones.One or more of these electronic devices can include a touch-sensitivesurface. The touch-sensitive surface can process multiple simultaneouspoints of input, including processing data related to the pressure,degree or position of each point of input. Such processing canfacilitate gestures with multiple fingers, including pinching andswiping.

When the disclosure refers to “select” or “selecting” user interfaceelements in a GUI, these terms are understood to include clicking or“hovering” with a mouse or other input device over a user interfaceelement, or touching, tapping or gesturing with one or more fingers orstylus on a user interface element. User interface elements can bevirtual buttons, menus, selectors, switches, sliders, scrubbers, knobs,thumbnails, links, icons, radio buttons, checkboxes and any othermechanism for receiving input from, or providing feedback to a user.

Example System Architecture

FIG. 14 is a block diagram of an example computing device 1400 that canimplement the features and processes of FIGS. 1-13. The computing device1400 can include a memory interface 1402, one or more data processors,image processors and/or central processing units 1404, and a peripheralsinterface 1406. The memory interface 1402, the one or more processors1404 and/or the peripherals interface 1406 can be separate components orcan be integrated in one or more integrated circuits. The variouscomponents in the computing device 1400 can be coupled by one or morecommunication buses or signal lines.

Sensors, devices, and subsystems can be coupled to the peripheralsinterface 1406 to facilitate multiple functionalities. For example, amotion sensor 1410, a light sensor 1412, and a proximity sensor 1414 canbe coupled to the peripherals interface 1406 to facilitate orientation,lighting, and proximity functions. Other sensors 1416 can also beconnected to the peripherals interface 1406, such as a global navigationsatellite system (GNSS) (e.g., GPS receiver), a temperature sensor, abiometric sensor, magnetometer or other sensing device, to facilitaterelated functionalities.

A camera subsystem 1420 and an optical sensor 1422, e.g., a chargedcoupled device (CCD) or a complementary metal-oxide semiconductor (CMOS)optical sensor, can be utilized to facilitate camera functions, such asrecording photographs and video clips. The camera subsystem 1420 and theoptical sensor 1422 can be used to collect images of a user to be usedduring authentication of a user, e.g., by performing facial recognitionanalysis.

Communication functions can be facilitated through one or more wirelesscommunication subsystems 1424, which can include radio frequencyreceivers and transmitters and/or optical (e.g., infrared) receivers andtransmitters. The specific design and implementation of thecommunication subsystem 1424 can depend on the communication network(s)over which the computing device 1400 is intended to operate. Forexample, the computing device 1400 can include communication subsystems1424 designed to operate over a GSM network, a GPRS network, an EDGEnetwork, a Wi-Fi or WiMax network, and a Bluetooth™ network. Inparticular, the wireless communication subsystems 1424 can includehosting protocols such that the device 100 can be configured as a basestation for other wireless devices.

An audio subsystem 1426 can be coupled to a speaker 1428 and amicrophone 1430 to facilitate voice-enabled functions, such as speakerrecognition, voice replication, digital recording, and telephonyfunctions. The audio subsystem 1426 can be configured to facilitateprocessing voice commands, voiceprinting and voice authentication, forexample.

The I/O subsystem 1440 can include a touch-surface controller 1442and/or other input controller(s) 1444. The touch-surface controller 1442can be coupled to a touch surface 1446. The touch surface 1446 andtouch-surface controller 1442 can, for example, detect contact andmovement or break thereof using any of a plurality of touch sensitivitytechnologies, including but not limited to capacitive, resistive,infrared, and surface acoustic wave technologies, as well as otherproximity sensor arrays or other elements for determining one or morepoints of contact with the touch surface 1446.

The other input controller(s) 1444 can be coupled to other input/controldevices 1448, such as one or more buttons, rocker switches, thumb-wheel,infrared port, USB port, and/or a pointer device such as a stylus. Theone or more buttons (not shown) can include an up/down button for volumecontrol of the speaker 1428 and/or the microphone 1430.

In one implementation, a pressing of the button for a first duration candisengage a lock of the touch surface 1446; and a pressing of the buttonfor a second duration that is longer than the first duration can turnpower to the computing device 1400 on or off. Pressing the button for athird duration can activate a voice control, or voice command, modulethat enables the user to speak commands into the microphone 1430 tocause the device to execute the spoken command. The user can customize afunctionality of one or more of the buttons. The touch surface 1446 can,for example, also be used to implement virtual or soft buttons and/or akeyboard.

In some implementations, the computing device 1400 can present recordedaudio and/or video files, such as MP3, AAC, and MPEG files. In someimplementations, the computing device 1400 can include the functionalityof an MP3 player, such as an iPod™.

The memory interface 1402 can be coupled to memory 1450. The memory 1450can include high-speed random-access memory and/or non-volatile memory,such as one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR). The memory 1450can store an operating system 1452, such as Darwin, RTXC, LINUX, UNIX,OS X, WINDOWS, or an embedded operating system such as VxWorks.

The operating system 1452 can include instructions for handling basicsystem services and for performing hardware dependent tasks. In someimplementations, the operating system 1452 can be a kernel (e.g., UNIXkernel). In some implementations, the operating system 1452 can includeinstructions for performing display buffer alignment. For example,operating system 1452 can implement the display buffer alignmentfeatures as described with reference to FIGS. 1-13.

The memory 1450 can also store communication instructions 1454 tofacilitate communicating with one or more additional devices, one ormore computers and/or one or more servers. The memory 1450 can includegraphical user interface instructions 1456 to facilitate graphic userinterface processing; sensor processing instructions 1458 to facilitatesensor-related processing and functions; phone instructions 1460 tofacilitate phone-related processes and functions; electronic messaginginstructions 1462 to facilitate electronic-messaging related processesand functions; web browsing instructions 1464 to facilitate webbrowsing-related processes and functions; media processing instructions1466 to facilitate media processing-related processes and functions;GNSS/Navigation instructions 1468 to facilitate GNSS andnavigation-related processes and instructions; and/or camerainstructions 1470 to facilitate camera-related processes and functions.

The memory 1450 can store software instructions 1472 to facilitate otherprocesses and functions, such as the display buffer alignment processesand functions as described with reference to FIGS. 1-13.

The memory 1450 can also store other software instructions 1474, such asweb video instructions to facilitate web video-related processes andfunctions; and/or web shopping instructions to facilitate webshopping-related processes and functions. In some implementations, themedia processing instructions 1466 are divided into audio processinginstructions and video processing instructions to facilitate audioprocessing-related processes and functions and video processing-relatedprocesses and functions, respectively.

Each of the above identified instructions and applications cancorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. The memory 1450 can includeadditional instructions or fewer instructions. Furthermore, variousfunctions of the computing device 1400 can be implemented in hardwareand/or in software, including in one or more signal processing and/orapplication specific integrated circuits.

To aid the Patent Office and any readers of any patent issued on thisapplication in interpreting the claims appended hereto, applicants wishto note that they do not intend any of the appended claims or claimelements to invoke 35 U.S.C. 112(f) unless the words “means for” or“step for” are explicitly used in the particular claim.

What is claimed is:
 1. A method comprising: detecting, by a computingdevice, movement of a graphical object from a first display buffercorresponding to a first display device to a second display buffercorresponding to a second display device; determining, by the computingdevice, a first path corresponding to the movement of the graphicalobject from a first location in the first display buffer to a secondlocation in the second display buffer; detecting, by the computingdevice, a correction in the first path associated with the seconddisplay buffer; and adjusting, by the computing device, a position ofthe second display buffer relative to the first display buffer based onthe detected correction.
 2. The method of claim 1, further comprising:fitting, by the computing device, a first curve to the first path;detecting, by the computing device, a discontinuity in a first portionof the first curve corresponding to the second display buffer; anddetermining, by the computing device, that an error exists in thealignment between the first display buffer and the second display bufferbased on the detected discontinuity in the first curve.
 3. The method ofclaim 1, further comprising: fitting, by the computing device, a secondcurve to a second portion of the first path corresponding to the firstdisplay buffer; projecting, by the computing device, the second curvetoward the second display buffer; adjusting, by the computing device, analignment of the second display buffer relative to the first displaybuffer such that the projected curve intersects the second location. 4.The method of claim 3, further comprising: determining, by the computingdevice, a second path corresponding to movement of a second graphicalobject from a third location in the first display buffer to a fourthlocation in the second display buffer; adjusting, by the computingdevice, the alignment of the second display buffer relative to the firstdisplay buffer based on the first path and the second path.
 5. Themethod of claim 3, further comprising: determining, by the computingdevice, a third path corresponding to movement of a third graphicalobject from a fifth location in the second display buffer to a sixthlocation in the first display buffer; adjusting, by the computingdevice, the alignment of the second display buffer relative to the firstdisplay buffer based on the first path and the third path.
 6. The methodof claim 1, further comprising: detecting, by the computing device, anew connection to the second display device; in response to detectingthe new connection to the second display device: generating, by thecomputing device, the second display buffer corresponding to the seconddisplay device, generating, by the computing device, a prompt in thesecond display buffer at the second location, causing, by the computingdevice the prompt to be presented on the second display device;receiving, by the computing device, user input with respect to theprompt at the second location; and adjusting, by the computing device,the position of the second display buffer relative to the first displaybuffer in response to receiving the user input.
 7. A non-transitorycomputer readable medium including a sequence of instructions that, whenexecuted by one or more processors, cause the processors to performoperations comprising: detecting, by a computing device, movement of agraphical object from a first display buffer corresponding to a firstdisplay device to a second display buffer corresponding to a seconddisplay device; determining, by the computing device, a first pathcorresponding to the movement of the graphical object from a firstlocation in the first display buffer to a second location in the seconddisplay buffer; detecting, by the computing device, a correction in thefirst path associated with the second display buffer; and adjusting, bythe computing device, a position of the second display buffer relativeto the first display buffer based on the detected correction.
 8. Thenon-transitory computer readable medium of claim 7, wherein theinstructions cause the processors to perform operations comprising:fitting, by the computing device, a first curve to the first path;detecting, by the computing device, a discontinuity in a first portionof the first curve corresponding to the second display buffer; anddetermining, by the computing device, that an error exists in thealignment between the first display buffer and the second display bufferbased on the detected discontinuity in the first curve.
 9. Thenon-transitory computer readable medium of claim 7, wherein theinstructions cause the processors to perform operations comprising:fitting, by the computing device, a second curve to a second portion ofthe first path corresponding to the first display buffer; projecting, bythe computing device, the second curve toward the second display buffer;adjusting, by the computing device, an alignment of the second displaybuffer relative to the first display buffer such that the projectedcurve intersects the second location.
 10. The non-transitory computerreadable medium of claim 9, wherein the instructions cause theprocessors to perform operations comprising: determining, by thecomputing device, a second path corresponding to movement of a secondgraphical object from a third location in the first display buffer to afourth location in the second display buffer; adjusting, by thecomputing device, the alignment of the second display buffer relative tothe first display buffer based on the first path and the second path.11. The non-transitory computer readable medium of claim 9, wherein theinstructions cause the processors to perform operations comprising:determining, by the computing device, a third path corresponding tomovement of a third graphical object from a fifth location in the seconddisplay buffer to a sixth location in the first display buffer;adjusting, by the computing device, the alignment of the second displaybuffer relative to the first display buffer based on the first path andthe third path.
 12. The non-transitory computer readable medium of claim7, wherein the instructions cause the processors to perform operationscomprising: detecting, by the computing device, a new connection to thesecond display device; in response to detecting the new connection tothe second display device: generating, by the computing device, thesecond display buffer corresponding to the second display device,generating, by the computing device, a prompt in the second displaybuffer at the second location, causing, by the computing device theprompt to be presented on the second display device; receiving, by thecomputing device, user input with respect to the prompt at the secondlocation; and adjusting, by the computing device, the position of thesecond display buffer relative to the first display buffer in responseto receiving the user input.
 13. A system comprising: one or moreprocessors; and a non-transitory computer readable medium including asequence of instructions that, when executed by the one or moreprocessors, cause the processors to perform operations comprising:detecting, by a computing device, movement of a graphical object from afirst display buffer corresponding to a first display device to a seconddisplay buffer corresponding to a second display device; determining, bythe computing device, a first path corresponding to the movement of thegraphical object from a first location in the first display buffer to asecond location in the second display buffer; detecting, by thecomputing device, a correction in the first path associated with thesecond display buffer; and adjusting, by the computing device, aposition of the second display buffer relative to the first displaybuffer based on the detected correction.
 14. The system of claim 7,wherein the instructions cause the processors to perform operationscomprising: fitting, by the computing device, a first curve to the firstpath; detecting, by the computing device, a discontinuity in a firstportion of the first curve corresponding to the second display buffer;and determining, by the computing device, that an error exists in thealignment between the first display buffer and the second display bufferbased on the detected discontinuity in the first curve.
 15. The systemof claim 7, wherein the instructions cause the processors to performoperations comprising: fitting, by the computing device, a second curveto a second portion of the first path corresponding to the first displaybuffer; projecting, by the computing device, the second curve toward thesecond display buffer; adjusting, by the computing device, an alignmentof the second display buffer relative to the first display buffer suchthat the projected curve intersects the second location.
 16. The systemof claim 9, wherein the instructions cause the processors to performoperations comprising: determining, by the computing device, a secondpath corresponding to movement of a second graphical object from a thirdlocation in the first display buffer to a fourth location in the seconddisplay buffer; adjusting, by the computing device, the alignment of thesecond display buffer relative to the first display buffer based on thefirst path and the second path.
 17. The system of claim 9, wherein theinstructions cause the processors to perform operations comprising:determining, by the computing device, a third path corresponding tomovement of a third graphical object from a fifth location in the seconddisplay buffer to a sixth location in the first display buffer;adjusting, by the computing device, the alignment of the second displaybuffer relative to the first display buffer based on the first path andthe third path.
 18. The system of claim 7, wherein the instructionscause the processors to perform operations comprising: detecting, by thecomputing device, a new connection to the second display device; inresponse to detecting the new connection to the second display device:generating, by the computing device, the second display buffercorresponding to the second display device, generating, by the computingdevice, a prompt in the second display buffer at the second location,causing, by the computing device the prompt to be presented on thesecond display device; receiving, by the computing device, user inputwith respect to the prompt at the second location; and adjusting, by thecomputing device, the position of the second display buffer relative tothe first display buffer in response to receiving the user input.